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I n t roduc t I on 



This report summarizes the work performed in the project 
entitled. "Design of Planning and Assessment System for the Division 
of Manpower and Institutions." and Introduces the substantive results 
of the project which are presented in the form of two separate ad~ 
vocate team reports that propose strategies for a planning and assess- 
ment system. 



The 

sect i ons : 


administrative summary which follows 


cons i s t s 


of five 


1 . 


Description of Original Proposal and 
Work, Including Personnel. 


Rev i sed 


Scope of 


1 1 . 


Selection of Advocate Team Members. 






III. 


Orientation Session. 






IV. 


Advocate Tear M riting Sessions. 






V. 


Recommenda t ions. 







I . Description of Original Proposal and Revised Scope of Work . 

On June 18, 1970, The Ohio State University Research Foundation 

submitted to the U.S. Office of Education a proposal initiated by 
Daniel L. Stufflebeam, Director of the Evaluation Center at Ohio 
State. The proposal was entitled, "Design of a Planning and Assess 
ment System for the Division of Manpower and Institutions (DM*)- 11 

The proposal . which was to span a pei of o.x and one~half 
months (June 15 to December 30, 1971) consisted of the following 
ob j ec t i ves : 

!. Completing a background study that would lay out the 
characteristics, needs, and opportunities of the new 
planning, management, and assessment system of the 
Division of Manpower and Institutions. 

2. Preparing a detailed specification for the design of 
the new system. 

3. Designing two alternative planning and assessment 
systems, complete with detailed specifications for all 
needed instrumentation. 

^ . Selecting or synthesizing the final system. 
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5. Completing pilot instrumentation according to the 
s pec i f i ca t 3 ons 

6. Orienting DM I staff to the new system. 

Procedures for achieving these objectives were elaborated into a 
series of twenty-seven steps; however, the funding for the initial 
proposal (which was to be followed by a continuation proposal) was 
requested only in terms of the first nine steps, as follows: 

1. Report #1 : c 1 a r i f i ca 1 3 on of background evaluation questions 

(needs for the new system, characteristics of present system) 

2. Confirmation of background evaluation questions 

3. Report ;/2: completion of background evaluation design 

4. Recruitment of Advocate and Convergence Teams 

5. Review of background evaluation design 

6. Implementation of the background evaluation 

7. Report #3 : {background evaluation findings 

8. Consultation as needed by the DM I staff regarding 

the implementation of their present evaluation system 
while a new one is under development 

9. Rep- » DM! evaluation objectives and alternatives 

evaluation specifications 

These steps were altered following a meeting with this National 
Center for Educational Research and Deve 1 opment /persiD^ie 1 when it 
^vas revealed that the time schedule (6-^ months) for scrnpl et i ng all 
27 steps would not result in an evaluation system that, could be imple- 
mented in time to service important decisions of NCEHD.. 

To meet NCERD needs, the time schedule and tasks were compressed 
into a two-month sequence. A brief description of tne revised scope 
of worrk for this grant of two months was presented i- letter to 
Ray Rackley, Program Monitor in DM!, and a copy of th s letter was 
also sent to Theresa Diorio of the Contract Office. Te pertinent 
section of this letter concerning the revised scope c the work is 
p rov i ded below. 

Essentially the scope of work for the twto 

months from June 28 to August 28 w/ill incorporate 

certain activities from the original proposal 
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submitted with the six end one-half month time 
frame. A brief description of activities involved 
is provided below. 

1. Members of DM1 will prepare the background 
study to be used by the advocate teams. 

2. OSU will recruit and orient two advocate teams. 

The orientation session will last three days. 

After an intervening period of time, the two 
teams will reconvene for a writing session of 
five days . 

3. The two evaluation strategies developed by 
the advocate teams will be presented to 

the Office of Education by September 1, 1971. 

It is understood that the amount of money for Phase 1 
(2 months) $24,440, will remain the same. It is 
also understood that any follow-up grants will take 
into consideration activities which have been completed 
during the first phase. 

Project personnel from The Ohio State University for hf 
revised scope of work included Daniel L. Stufflebeam, Project 
Supervisor (10% time for two months); Diane Reinhard, Project 
Manager, (100% time for two months); and a secretary (100% time 
for two months ) . 

II- Selection of Advocate Team Members . The revised scope of work 
called for recruiting two advocate teams, each of which would 
separately develop an evaluation design for DM I . 

Michael Scriven and Daniel L. Stufflebeam were solicited and 
each agreed to chair one of the teams. The composition of their 
teams included: 

Michael Scriven, Chairman 

University of California at Berkeley 

Gene V. Glass 

University of Colorado 

Wells H i vel y 

University of Minnesota 

Robert E. Stake 

University of Illinois 
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Daniel L. Stufflebeam, Chairman 

The Ohio State University 

Henry M . B r i c ke 1 1 

Institute for Educational Development 

New York, New York 

Egon G. Guba 

Indiana University 

William B . Mi chae 1 

University of Southern California 

The teams are noteworthy for the high caliber professional 
people who agreed to serve. 

Since the evaluation system for which these two teams were 
to devise a strategy concerned the Educational Labs and R&D Centers, 
it was decided that a rep resenta t i ve from the labs and from the 
R&-D Centers should be invited to serve as consultants to the advocate 
teams . 



John L. Holland of the Center for Social Organization of Schools 
and Richard E. Schutz of the Southwest Regional Laboratory for Educa- 
tional Research and Development agreed to serve as consultants to the 
advocate team chaired by Michael Seri ven. Max G. Abbott of the Center 
for the Advanced Study of Educational Administration and Robert S. 
Randall of Southwest Educational Development Laboratory consented to 
serve as consultants for the advocate team chaired by Daniel 
S tu f f 1 ebeam . 

III. Orientation Session. On July 24, 1971 , an orientation session 
was held in Washington, D.C., for all advocate members and consultants. 
The purpose of this meeting was to provide background data and 
answer any questions team members or consultants had regarding their 
task. While a three day orientation session had been projected, 
feasibility constraints allowed only a one-day session. 

In planning for this session, the project manager at OSU and the 
project monitor from the Office of Education worked closely together. 
They compiled questions that advocate team members wished to have 
answered at the session and forwarded them to relevant persons within 
NCERD. Advocate team members were also asked to identify documents 
needed to assist them in their task of developing evaluation strategies. 

A compilation of documents resulted from the orientation session 
planning effort, and each advocate team member was given a packet of 
materials which included: 

1 . Lab and Center Background : 

Schmidtlein, Frank. A Status Report on the 

National Program of Educational Laboratories, 




6 



5 



R&D Centers and Related RCD Institutions 
Administered by the Division of Manpower 
and 1 ns t i t u t i ons . April 12, 1S71. 

Brickell, Henry M. Testimony Before the Sub- 
committee on the Department of Health , 
Education and Welfare, Senate Appropria - 
tions Committee, United States Senate. 

M a rch 30, 1971. 

Chase, Francis. E ducational Research and Devel - 
opment in the Sixties: The Mixed Report Card . 

Submitt d to the Select Subcommittee on 
Education, L.S. House of Representatives. 
April, 1971. 

2. Recent and current reporting guidelines an d _PGJS 
I n forma t i on : 



Centers : 



Guidelines for FY 71 Program Plan and 
Budget Request (PRDB) . Jury 24, 1970. 

Annual Report Guidelines . June 1 , 1970. 

Guidelines for Qua rter.ly. Prog res s Reports . 
May 13, 1970. " 

La bora t o r i es : 

Program Guidelines: Contractor's Request 

for Continued Funding (CRCF) July 1970 
Rev i s i on . 

Program Guidelines: Basic Program Plans . 

July 1 , 1969. 

National Program on Early Childhood Education: 

Contractor's Request for Continued Fund - 
i ng . June 25, 1970. 

Instruction's for September 15 Report Budget 
Mat er i a 1 s . August 5, 1970. 

Centers and Laborat or i es : 



DRAFT: Information and Instructions 

for submittal of the Basic Program 
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Plan; Educational Laborator i es t Research 
and Development Centers , June 17, 1971- 

DRAFT: FV 1972 Interim Guidelines fo r 

Annual Budget Justification (AB J)_;. _P jy_ 
is ion of Manpower ajid_ I ns t i t u t i on_s_ 9 
July 17, 1971 - 



PG1S : 



Viehoever, Kent. Posj t ion Pager on the 
Project Grant Information System (PGj$_I 
and the Division of Manpower ari d 
I ns t I tut ions . April 14, 1971- 



3. Documents furnished to DM 1 by the Centers and Labor- 
atories in response to the guide! ines or paragraph 3 (above): 

The most recent Basic Program P 1 a n ^ Contractor's 
Request for Continued Funding , or Program Pla_n 
and Budget Request from each of the institutions 
sponsored by DM I * 

4. Papers relating to the contract for the design of a new 
planning and assessment system for DM!: 

Stuff 1 ebeam , Daniel L. Proposal: Des iqn of a 

Planning and Assessment System for the Division 
of Manpower and Institutions . June 18, 1971* 

Rackley, W. Ray and Reinhard, Diane L. Charge, _to 
the Advocate Teams . Design of a Planning and 
Assessment System,. for the Division of Manpower 
and 1 ns t i tut i ons . July 9, 1970* 

Rackley, W. Ray. Memo to Dr. Silbermant Ma jor 
Steps in the Design of the DM I Evaluation System. 

June 29 , 1 97 1 - 

5 . Selection of information on product evaluation. 

6 . Selection of information used to guide site visit teams. 

A charge to the advocate team was also developed* It reads as 
f o 1 1 ows : 

G i ven 



1. Criteria for an institutional support 
policy and associated assessment pro- 
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cedurcs for the Division of M a n p ow er and 
I ns t i t u t i ons (Cf. Frye, June 3 , 1971 jPP-3~5), 

2 . A support po 1 icy and assoc i3ted assessment 
o roc edu res which meet the criteria (Cf. 

Frye, above), and 

3. A statem ent of the ten types of decisions 

included in the new support policy and 
associated assessment procedures (C f . 

Rackley, June 18, 1 97 1 , pp. 2^4), 

des 1 gn an assessment system which will enable DM I leaders to 
make responsible decisions in regard to the institutions and 
programs which they support. 

The design will specify all steps and procedures leading up 
to each of the ten decisions, including: 

1. A statement of the criteria to be used in making 
the dec i s i ons , 

2. A description of the assessment information needed 
to make the specified decisions, 

3. A statement of how the information will be obtained, 
processed, interpreted and used by DM 1 decision 
makers , 

4. A statement of who will participate in the assessment 
activities, including the responsibilities and 

qua 1 i f i ca t i ons needed for each position, 

5. Outlines or specifications of instruments to be 
used , 

6. A statement of the conceptual, precedural, and 
orga n i zat i ona 1 unity or integration of the ten 
types of reviews or decisions, 

7- A projected budget, and 

8 . Recommend at ions for install i ng the s ys t em , including 
completion of instruments, training of staff, etc. 

The report submitted by each team will include, further, 
a narrative walk-through of how each of the ten reviews is 
to be conducted. 
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The orientation session itself began with the presentation by 
Harry Silberman, Associate Commissioner of NCERD. This was followed 
by a discussion of the “Support Policy 11 paper dated June 3, 1971, with 

Chuck Frye answering questions that were posed concerning it. The 
remainder of the morning and the first part of the afternoon were 
spent in discussion or questions generated by the advocate team members- 
Questions were addressed to the following OE personnel who attended 
the session: 

Charles Frye, Director, DMl 
Greta Gibson, Program Specialist 
Sarah Gideonse, Program Specialist 
Susan Gruskin, Program Specialist 
Bruce Hawki nson , Product Diffusion Officer 
Joseph Lipson, Program Planner, Commissioner's Planning 
Unit for the National institute of Education 
Ward Mason, Chief, Program Operations Branch, DMl 
Ernest Michelson, Deputy Director, DM I 
Ray Rackley, Leader, Program Review Team 
Harry Silberman, Associate Commissioner, NCERD 
Kent Viehoever, Leader, Reports and Analysis Team 

In addition to the NCERD persons, Dr. Francis Chase, an 
authority on Educational Laboratories and R&D Centers, served 
a C- a consultant for the day. 

During the latter half of the afternoon, each advocate team met 
separately to discuss individual concerns and talk with Office of 
Education resource persons. 

All members of the teams and the consultants were able to attend 
this session except Robert S*ta ke , who had a prior commitment. Therefore, 
Diane Reinhard traveled to CIRCE at the University of Illinois on 
August 10 to orient him by answering questions and providing documents. 

Finally, transcripts of the orientation session were sent to 
advocate team members and their consultants prior to their convening 
for the writing sessions. Team members also received a sample of site 
visit reports which had had identifying information on names of insti- 
tutions and site reviewers deleted. 

IV. Advocate Team Writing Sessions . The chairman of the advocate teams 
were permitted to select the site for their conference writing session. 
They were offered the services of a technical writer or stenographer 
and they were provided any needed equipment (typewriters, dictating 
machines. Xerox facilities) at the site of their meeting. Each team 
was also allocated an amount of money to be spent following the writing 
session for editorial work or additional consultant time for the 
chairman of the teams. 
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The advocate team chaired by Daniel Stuff i ebeam met August 
5-9 at Phoenix, Arizona, Del Webb Towne House. The product of this 
meeting is presented in the accompnaying Advocate Team Report entitled 
"Design for Evaluating institutions and Programs. 11 

The advocate team chaired by Michael Scriven met for its 
writing session, August l8~21 at Santa Barbara, Cal ifornia, Santa 
Barbara Biltmore. The product of this meeting is presented in the 
accompanying report entitled "An Evaluation System for Labs and 
RS-D Centers . " 

Both Project Manager Diane Reinhard and Project Monitor Ray 
Rackley attended both sessions. Ray Rackley brought additional 
documents (ie., reports from all of the institutions) for advocate 
team members to utilize if they wished. 

Since the idea of utilizing advocate teams was for the purpose 
of obtaining two different strategies, special care was taken by 
Reinhard and Rackley not to have the proceedings of the first advocate 
team influence or contaminate in any way the proceedings of the second 
s es s i on . 

V. Recommendat ions . First, it is recommended that the decision 
makers in Division of Research and Development Resources (previously 
known as DM l ) , NCERD, obtain judgements of the two strategies 
generated by the two advocate teams from an external group. 

Second, it is furthur recommended that the chairman of each 
advocate team be summoned to Washington to answer any questions that 
the decision makers may personally have regarding each strategy. 

Third, following a decision between the strategies, it is 
recommended that DRDR attempt in every way possible to put into 
operation the selected strategy. 




11 



